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Description 

This invention pertains to mobile robots or 
automatic guided vehicles (AGVs) and more par- 
ticularly, to computer control and processing of 
visual images from a CCD camera In a computer 
controlled robotic arm. 

DESCRIPTION OF THE RELATED ART 

Present day assembly processes, such as 
automated flexible manufacturing systems or cells 
for manufacturing small, delicate parts such as 
integrated circuits, require that accurate workpiece 
transfer takes place between an automatic guided 
vehicle (hereinafter, AGV) and an assembly or pro- 
cess machine, or a workpiece storage location. The 
problem is highlighted in those systems in which 
fully automatic operation without human interven- 
tion is desired to complete the workpiece transfer. 
Typically, present day material handling systems 
require a high degree of docking or parking accu- 
racy for such workpieces to be transferred suc- 
cessfully without human assistance. 

Particularly where small workpieces are in- 
volved, the typical AGV Is not able to ascertain the 
precise location of a workpiece prior to transferring 
it automatically between the AGV and a process 
machine or storage location. 

Also desirable for successful workpiece trans- 
fer is for the presence or absence of a workpiece 
to be detectable. That is, if there is no workpiece at 
a process machine, that fact should be ascertain- 
able automatically. 

Related to the problem of accurate workpiece 
transfer is the task of ensuring accurate parking of 
an AGV. Often, stops or cone shaped devices are 
used for mechanically guiding an AGV into a dock- 
ing or parking position. These mechanical devices 
present the disadvantages of vibrating the AGV and 
of finding a suitable place in which to locate and 
mount them. 

In EP-A-0 120 197, there is described an op- 
tical track-following apparatus for an Industrial robot 
according to the first part of claim 1. The apparatus 
Includes means for illuminating a line to be fol- 
lowed or for producing a light-emitting line to be 
followed, an image of the line being focused on to 
a light sensitive panel from which correction signals 
for the robot are derived by a computer. 

It is an object of the present invention to pro- 
vide an improved robot system. 

According to the present invention there is 
provided a visually guided robot system for trans- 
ferring workpieces between work sites on work 
stations, wherein a vision device is provided on a 
robot arm to provide location information to the 
roljot control system, characterized In that 



the vision device Is attached to the robot arm 
adjacent to a robot arm gripper assembly, 

and in that the system further includes a nest 
means for holding a workpiece at a work station 

5 surface; and 

a pair of beacon devices located adjacent the 
nest means on a work station surface for viewing 
by said vision device to enable the robot system to 
determine the location of the work site on the work 

10 station. 

One example of the invention provides for a 
CCD camera mounted on the gripper end of a 
flexible, workpiece-manipulating, robot arm, which 
is in turn mounted upon an AGV or mobile robot. 

75 The gripper end of the robot arm includes a com- 
puter controllable gripper for grasping and releas- 
ing workpieces. The camera is rigidly mounted 
adjacent to the gripper and has fixed coordinates In 
the gripper coordinate system, or frame of refer- 

20 ence. The invention also provides for at least two 
active or passive locating beacons in known posi- 
tions with respect to the workpiece nest on a 
process machine. The beacons are detected by the 
camera and the visual information is then sent to a 

25 control computer on board the AGV, which pro- 
cesses the information and guides the robot arm In 
picking up or depositing workpieces. The visual 
information also can be used to guide the robot 
arm in pushing a switch located on the process 

30 machine, for the purpose of letting the process 
machine know that the AGV is ready to load or 
unload a workpiece. 

During a workpiece transfer sequence, the ro- 
bot arm is guided with high precision to a work- 

35 piece nest, from which a workpiece is picked up or 
deposited, whichever the case may be. The high 
accuracy is achieved by use of the CCD camera, 
which provides visual information to the AGV com- 
puter about the position of the beacons with re- 

40 spect to the gripper end of the robot arm. This 
aspect of the invention is advantageous over prior 
art techniques in that the CCD camera being used 
as a guidance sensor is located adjacent to the 
gripper, giving a greatly expanded scale of view of 

45 the beacons with respect to the gripper. Also, be- 
cause the camera is fixed In the gripper coordinate 
system, the camera always points in the same 
direction relative to the gripper, no matter how the 
gripper end of the robot arm Is tilted or rotated. 

50 Thus, the targeted beacons can always be kept in 
view as the gripper moves toward the workpiece 
nest. This high degree of manipulative accuracy 
allows the robot arm to transfer very small parts, 
such as are associated with Integrated circuit man- 

55 ufacture. Fully automatic parts transfer is, therefore, 
possible without necessity of human assistance. 

Because the robot arm gripper end can be 
accurately positioned anywhere within the reach of 
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the arm, the AGV docking accuracy requirements 
correspondingly can be less. This is advantageous 
over prior art systems in which an AGV must be 
positioned with greater accuracy to effect success- 
ful load transfers. Not only is greater parking accu- 
racy more difficult to achieve technically, but it is 
more expensive as well, since costlier guidance 
equipment must be used. 

One of the embodiments of the invention in- 
cludes a beacon located in the workpiece nest. 
From the perspective of the CCD camera in the 
robot arm, the view of the beacon is blocked when 
a workpiece is present In the workpiece nest, and 
Is not blocked when a workpiece is not present. 
Thus, the control computer can determine whether 
or not a workpiece is present in the nest. This 
feature prevents one workpiece from being placed 
upon another already In the nest, and prevents the 
AGV from attempting to pick up a workpiece which 
is not there. The task of material tracking is simpli- 
fied as well, since the control computer can inform 
a supervisory computer that a workpiece is already 
at the designated process machine or that an ex- 
pected workpiece is not there. 

Another embodiment of the invention utilizes 
the robot arm and CCD camera to achieve greater 
parking accuracy than may be obtainable from the 
primary navigation system. In this embodiment, the 
computer on board the AGV responsible for naviga- 
tion receives data from the control computer. When 
the AGV gets within robot arm's length of the 
expected position of, the desired parking location, 
the robot arm and camera search for the locating 
beacons on the process machine or other parking 
place and when they are found, the AGV navigation 
computer is able to determine with precision the 
AGV location with respect to the desired parking 
place. The guidance system then can accurately 
bring the AGV into the desired parking position. 

In a further embodiment, the robot arm Is used 
to press a switch located on a process machine, to 
inform the process machine that a workpiece trans- 
fer is desired. A beacon Is located near the switch 
and is for indicating the location of the switch to 
the CCD camera which, in cooperation with the 
control computer, guides the gripper end of the 
robot arm into sufficiently forceful contact with the 
switch to trip it. This beacon also is useful for 
signaling to the AGV as a part of a handshaking 
sequence. 

One of the advantages of the Invention is in 
providing a single structure, namely, the robot arm 
and CCD camera, to effect more than one of the 
above features of the Invention. 

A further advantage to using the CCD camera 
Is its small size and weight, which Is an important 
consideration when, as here, the camera Is moun- 
ted upon the robot arm. 



These objects and advantages of the Invention 
will be apparent from a reading of the specification 
in conjunction with the figures. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

Rgure 1 is a perspective view showing the 
gripper end of the robot arm with the CCD camera 
mounted just above the gripper, and showing a 
10 workpiece nest and beacons on a process ma- 
chine. 

Figure 2 is a side view showing an AGV with 
its robot arm manipulating a workpiece on a pro- 
cess machine. 
75 Figure 3 is a system level block diagram. 

Figure 4 is a block diagram Illustrating the 
vision calibration mode. 

Figure 5 is a block diagram illustrating the 
teach mode. 

20 Figure 6 Is a block diagram illustrating the 

playback mode. 

Figure 7 is a block diagram illustrating the 
subroutine used by the teach and playback modes 
to calculate centroids. 
25 Figure 8 shows the windows used in calculat- 

ing centroids. 

Figure 9 Is a block diagram illustrating the 
handshake sequence between an AGV and a pro- 
cess machine to initiate workpiece transfer between 
30 the AGV and the process machine. 

Figure 10 is a block diagram illustrating the 
handshake sequence between an AGV and a pro- 
cess machine at the termination of workpiece trans- 
fer between the AGV and the process machine. 
35 Figure 1 1 Is a block diagram showing the con- 
trol computer, CCD camera and robot arm. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

40 

As seen in Figure 1, a charge coupled device 
(CCD) camera 1 is mounted upon a robot arm 
gripper end 2. A vision cable 3 extends from the 
CCD camera back along the robot arm 4 to a 

45 general purpose computer (referred to herein as 
the control computer 24, shown in Figure 11) via 
interface circuitry 25 well known in the art. The 
robot arm 4 is connected to the control computer 
24 via interface circuitry 25, as shown In Rgure 1 1 . 

50 Also mounted upon the robot arm gripper end 2, 
immediately subjacent to the CCD camera 1, Is a 
gripper assembly 5 which includes opposing grip- 
per fingers 6. The gripper fingers 6 are slidably 
mounted in an actuation box 7 in which there is a 

55 motor controlled by the control computer 24 to 
actuate a set of gears to slidably move simulta- 
neously the fingers 6 together or apart, In the 
direction of the arrows 8 or 9 respectively. Mounted 
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on each finger 6 is workpiece sensor 10 to sense 
the presence or absence of a workpiece, such as 
the workpiece 1 1 shown in Rgure 2. Preferably, the 
sensors 10 are well known optical sensors or prox- 
imity detectors. 

As seen in Figure 2, the robot arm 4 Is mul- 
tljointed which gives it great flexibility. Joints can 
be seen at 20, 21 . 22 and 23. Joint 23 is a swivel 
joint which allows the robot arm gripper end 2 to 
rotate about its axis. 

Turning now to Figure 2, an AGV 12 is shown 
upon which is mounted a robot arm 4 with a 
workpiece 11 held by the gripper fingers 6. The 
robot arm 4 and workpiece are shown in a position 
in which the robot arm can either deposit the 
workpiece by moving the gripper fingers 6 apart, or 
remove the workpiece by moving the fingers 6 
together and retracting the robot arm toward the 
AGV 1 2. To deposit the workpiece requires spread- 
ing the fingers 6 until both of the the workpiece 
sensors 10 Indicate to the control computer 24 that 
they no longer detect the presence of the work- 
piece 11. Similarly, removing the workpiece 11 
from the process machine 13 requires that the 
fingers 6 be moved toward each other a sufficient 
distance to cause both workpiece sensors 10 to 
detect the presence of the workpiece. 

Referring back to Figure 1, a pair of locating 
beacons 14 Is positioned in the upper surface 15 of 
the process machine 13, near a workpiece nest 16. 
The workpiece nest 16 is for holding a workpiece 
while operations are carried out upon It by the 
process machine. The two locating beacons 14 
preferably are a pair of light emitting diodes (LEDs) 
and are detectable by the CCD camera 1 . 

After the locating beacons 14 have been de- 
tected by the CCD camera 1, the control computer 
24 processes the visual information from the cam- 
era 1, and from it determines positions of the 
locating beacons 14 with respect to the robot arm. 
The control computer 24 has stored in Its memory 
the expected position of the workpiece nest with 
respect to the locating beacons, thus, after the 
control computer 24 knows the position of the 
locating beacons with respect to the robot arm, it is 
simple geometry to calculate the position of the 
workpiece nest with respect to the robot arm. With 
the position of the workpiece nest relative to the 
robot arm 4 known, the control computer 24 knows 
with great precision where to direct the robot arm 
in order to deposit or remove workpieces from the 
process machine 13. 

One embodiment of the invention includes a 
nest beacon 17 located within a workpiece nest 16 
as shown in Figure 1. Preferably, this beacon also 
is an LED. The nest beacon 17 Is visible to the 
CCD camera 1 when a workpiece Is not present In 
the nest, and the nest beacon Is not visible when 



the workpiece is in the nest. Consequently, prior to 
beginning a load/unload sequence, the control 
computer 24 can quickly determine whether or not 
a workpiece Is already present In the nest by 

5 directing the robot arm to train the CCD camera 1 
on the expected position of the nest. 

Another embodiment of the invention includes 
a pushbutton switch 18 and closely adjacent switch 
beacon 19, both mounted on the upper surface 15 

70 of the process machine 13, as shown In Figure 1. 
Preferably, the switch beacon 1 9 Is an LED. Ideally, 
the process machine 13 is computer controlled, 
and the switch beacon 19 and pushbutton switch 
18 are used for positive handshaking between the 

75 AGV and process machine by pressing the switch 
with the robot arm gripper assembly 5 and monitor- 
ing the on/off condition of the beacon with the CCD 
camera 1 . 

The control computer 24 locates the pushbut- 
20 ton switch 18 by finding the switch beacon 19 with 
the CCD camera 1. The gripper assembly 5 is 
sufficiently large and the pushbutton switch 18 
close enough to the switch beacon 19 to allow the 
control computer 24 simply to direct the gripper 
25 assembly toward the switch beacon and be as- 
sured of coming into contact with the pushbutton 
switch. 

It Is obvious that infrared (IR) LEDs could be 
used to perform the handshake sequence rather 

30 than using the LED/pushbutton-switch combination. 
In a similar vein, IR LEDs, or any other wavelength, 
could be used for all the beacons. IR is attractive 
since it is well known that typical CCD cameras are 
most sensitive to IR wavelengths, however, LEDs In 

35 the visible spectrum are preferred since it is easier 
to spot malfunctioning units. 

It is obvious that the locating beacons 14 can 
also be provided for storage locations as well as for 
the processing locations on the process machines. 

40 Thus, the AGV could store and retrieve parts from 
storage. 

SYSTEM OPERATION 

45 Turning now to Figure 3, an overall system 
view of the AGV workpiece transfer system Is 
shown. After the AGV powers up at 100, the control 
computer 24 transfers from bubble memory, or 
other nonvolatile mass storage, the software neces- 

50 sary to run the AGV workpiece transfer system, as 
seen at 101. At 102 the decision is made by a 
human operator whether or not to calibrate the 
vision system. If the vision system has never been 
calibrated or if recall bration is required due to 

55 anything which could have caused rotation of the x- 
y plane of the CCD camera coordinate system 
about the optical axis of the CCD camera 1, then 
the operator will choose to calibrate the vision 
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system at 103. 

The vision calibration nnode is Illustrated in 
Figure 4. The purpose of this mode Is to find the 
relationship between the CCD camera x-y coordi- 
nate system and the g ripper x-y coordinate sys- 5 
tem. The two x-y planes can be made parallel by 
mechanical alignment, however, it Is more difficult 
to mechanically rotate the camera 1 to make sure 
that the two x-axes are parallel and the two y-axes 
are parallel. Thus, any rotational angle between the io 
two sets of axes is found by the vision calibration 
mode 103. and coordinates of objects seen by the 
CCD camera 1 are adjusted mathematically by the 
control computer 24 to be In terms of the gripper 
coordinate system. Because both the camera 1 and 75 
gripper assembly 5 are mounted on the same 
structure (the robot arm gripper end 2), the rela- 
tionship between the two coordinate systems re- 
mains fixed as long as none of the component 
parts of the gripper assembly or CCD camera are 20 
changed or otherwise disturbed. 

The overall frame of reference for the AGV 
workpiece transfer system is the AGV base coordi- 
nate system, and it is in this coordinate system that 
the control computer 24 works when it calculates 25 
the positions of the various structures, e.g., work- 
piece nest 16, located on the process machine 13. 
Therefore, the coordinates of objects detected by 
the CCD camera 1 must eventually be put in terms 
of the base coordinate system. The mechanical 30 
geometry of the robot arm is well known to the 
control computer 24, so, the coordinates of objects 
in the gripper coordinate system can be mathemat- 
ically transformed by the control computer into 
coordinates in the AGV base coordinate system. 35 
Thus, by determining in vision calibration mode 
103 the mathematical relationship between the 
CCD camera coordinate system and the gripper 
coordinate system, the control computer 24 can 
mathematically transform coordinates from the 4o 
CCD camera coordinate system into coordinates in 
the AGV base coordinate system. 

Returning to Figure 3, the human operator de- 
cides at 104 whether or not to enter the teach 
mode 105. The purpose of the teach mode 105 is 45 
to find and store In the control computer 24 mem- 
ory, the AGV base coordinates of each beacon, 
and the x, y and theta offsets for the workpiece 
nests, for each of the different logical process 
machine layouts of beacons and workpiece nest. 50 
Theta is an angle measured from a line parallel to 
the x-axis in the x-y plane. As an example, in a 
fiexible manufacturing cell with ten process ma- 
chines, there may be only three logical layouts 
involved. That is, more than one machine will have 55 
the same arrangement of beacons and workpiece 
nest with the same geometrical spacings and an- 
gles between the respective beacons and work- 



piece nest. Some of these logical layouts will have 
a pushbutton switch 18 associated with one of the 
beacons 19, and one embodiment of the Invention 
includes a nest beacon 17 inside each workpiece 
nest 16. The teach mode 105 is illustrated in Rg- 
ures 5, 7 and 8. 

Turning again to Figure 3, the human operator 
decides at 1 06 whether or not to enter the autorun 
mode 107. The autorun mode Is the working mode 
for the AGV workpiece transfer system in which the 
AGV Is transferring workpieces to and from various 
process machines or storage locations. The auto- 
run mode is illustrated in Figures 6, 7 and 8. 

Figure 9 illustrates the handshake sequence 
between an AGV 12 and a process machine 13 
which takes place prior to workpiece transfer. One 
of the purposes of the handshake Is to Inform both 
the AGV and the process machine that the other is 
ready to begin workpiece transfer. Figure 10 illus- 
trates the handshake sequence which takes place 
between the AGV and the process machine after a 
workpiece has been successfully transferred. One 
of the purposes of the ending handshake Is to let 
both the AGV and process machine know that the 
workpiece transfer has concluded and each of the 
parties to the transfer can now proceed with other 
tasks. A further purpose served by both handshake 
sequences is relevant where a central computer is 
scheduling work, keeping track of material fiow, 
and controlling the AGV and process machines. 
That purpose is that the central computer Is in- 
formed in a positive way when work is, and Is not, 
successfully started and completed. 

Claims 

1. A visually guided robot system for transferring 
workpieces between work sites on work sta- 
tions, wherein a vision device (1) is provided 
on a robot arm to provide location information 
to the robot control system, characterized in 
that 

the vision device Is attached to the robot 
arm adjacent to a robot arm gripper assembly 
(5). 

and in that the system further includes a 
nest means (16) for holding a workpiece at a 
work station surface (1 5); and 

a pair of beacon devices (14) located adja- 
cent the nest means (16) on a work station 
surface (15) for viewing by said vision device 
(1) to enable the robot system to determine 
the location of the work site on the work sta- 
tion. 

2. The visually guided robot system according to 
claim 1 , further characterized by an additional 
beacon device (17) located within said nest 
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means. 

3. The visually guided robot system according to 
claim 1, further characterized by handshake 
device (18) to enable the robot control system 
to know when a workpiece transfer is to begin 
and is to end. 

4. The visually guided robot system according to 
claim 3, characterized by a further beacon 
device (19) located adjacent to said handshake 
device (18). 

5. The visually guided robot system according to 
claim 1, characterized in that the robot arm 
gripper assembly has gripper fingers (6) and 
there is provided a sensor device (10) located 
in each gripper finger (6) to determine when 
the gripper assembly (2) is holding a work- 
piece. 

Revendications 

1. Systeme de robot a guidage visuel pour trans- 
ferer des pieces a usiner entre des sites d'usi- 
nage sur des postes de travail, dans lequel un 
dtspositif de vision (1) est i\x6 sur un bras de 
robot afin de fournir une information de posi- 
tion au systeme de commande du robot, ca- 
ract^ris^ en ce que le dispositif de vision est 
fix^ au bras de robot de manidre contigue a un 
ensemble pr^henseur de bras de robot (5), et 
en ce que le systeme comprend en outre un 
dispositif receptacle (16) pour maintenir une 
pidce a usiner a la surface d'un poste de 
travail (15). et une paire de dispositifs de bali- 
sage (14) situ^s a cote du dispositif receptacle 
(16) a la surface d'un poste de travail (15) afin 
de visionner au moyen dudit dispositif de vi- 
sion (1) pour permettre au systeme de robot 
de determiner I'emplacement du site d'usinage 
au poste de travail. 

2. Systeme de robot a guidage visuel selon la 
revendication 1, caracterise en ce qu'il com- 
prend en outre un dispositif de balisage sup- 
piementaire (17) situe a I'interieur dudit dispo- 
sitif receptacle. 

3. Systeme de robot a guidage visuel selon la 
revendication 1, caracterise en ce quNI com- 
prend en outre un dispositif d'etablissement de 
liaison (18) pour permettre au systeme de 
commande de robot de determiner quand un 
transfert de piece a usiner doit commencer et 
quand il doit se terminer. 

4. Systlme de robot a guidage visuel selon la 



revendication 3. caracterise en ce qu'il com- 
prend en outre un dispositif de balisage sup- 
piementaire (19) situe a proximite dudit dispo- 
sitif d'etablissement de liaison (18). 

5 

5. Systdme de robot a guidage visuel selon la 
revendication 1, caracterise en ce que Ten- 
semble prehenseur du bras de robot a des 
doigts prehenseurs (6) et en ce qu*il est prevu 
70 un dispositif detecteur (10) situe dans chaque 

doigt prehenseur (6) afin de determiner quand 
I'ensemble prehenseur (2) tient une piece a 
usiner. 

15 Patents nsprij Che 

1. Ein sichtgefuhrtes Robotersystem zur Uberfiih- 
rung von Werkstucken zwischen Bearbeitungs- 
stellen auf Bearbeitungsstationen. worin eine 
20 Sichtvorrichtung (1) an einem Roboterarm vor- 

gesehen ist, um dort Informationen an das 
Roboter-Steuersystem zu liefern, dadurch ge- 
kennzeichnet, dafi 

die Sichtvorrichtung an dem Roboterarm 
25 angrenzend an eine Roboterarm-Gretfeinheit 

(5) befestigt ist, 

und 6aB das System ferner eine Aufnah- 
meeinrichtung (16) zum Halten eines Werk- 
stucks an einer Oberflache (15) der Bearbei- 
30 tungsstation umfaiJt; und 

zwei Leitstrahl-Vorrichtungen (14) angren- 
zend an die Aufnahmeeinrichtung (16) an einer 
Oberflache (1 5) der Bearbeitungsstation vorge- 
sehen sind, um mittels der Sichtvorrichtung (1) 
35 eine Beobachtung durchzufuhren, um dem Ro- 

botersystem die Bestimmung des Ortes der 
Bearbeitungsstelle an der Bearbeitungsstation 
zu ermoglichen. 

40 2. Das sichtgefuhrte Robotersystem nach An- 
spruch 1, ferner gekennzeichnet durch eine 
zusatzliche Leitstrahl-Vorrichtung (17), die in- 
nerhalb der Aufnahmeeinrichtung angeordnet 
ist. 

45 

3. Das sichtgefuhrte Robotersystem nach An- 
spruch 1, ferner gekennzeichnet durch eine 
Annahme-Bestatigungsvorrichtung (18), die es 
dem Roboter-Steuersystem ermbglicht. Kennt- 

50 nis davon zu eriangen, wann eine WerkstLick- 

ubergabe beginnen und wann sie enden soil. 

4. Das sichtgefuhrte Robotersystem nach An- 
spruch 3, gekennzeichnet durch eine weitere 

55 Leitstrahl-Vorrichtung (19), die angrenzend an 

die Annahme-Bestatigungsvorrichtung (18) an- 
geordnet ist. 
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5. Das sichtgefuhrte Robotersystem nach An- 
spruch 1, dadurch gekennzeichnet, daB die 
Roboterarm-Greifeinheit mit Greiffingern (6) 
versehen ist und eine Sensorvorrichtung (10) in 
jedem Greiffinger (6) angeordnet ist, um fest- s 
zustelten. daB die Greifeinheit (2) ein Werk- 
stuck halt. 
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COMPUTE X.ANO Y 

POSITION OF EACH 

BEACON IN AGV 

BASE COORDINATE 
SYSTEM 
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ALIGN GRIPPER 
WITH THE 
BEACONS 
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TEACH X.Y.THETA 
OFFSETS OF WORK PIECE 

NEST FROM GRIPPER 
AS ALIGNED 




Fig. 5 
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TO THE PROCESS 
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CONTROL COMPUTER 



I 
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PERFORM 
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AREA (NEST) IS 
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Fig. 7 
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IS LEAVING 





Fig. 10 



I 
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